Papers by Zayd Muhammad Kawakibi Zuhri
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax (2026.findings-acl)
Copied to clipboard
| Challenge: | Quantized models using softpick outperform softmax on standard benchmarks . softmax is widely used in statistics and especially in machine learning . |
| Approach: | They introduce a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. |
| Outcome: | The proposed model outperforms softmax on benchmarks with lower bit precisions. |
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-Layer Key-Value (MLKV) sharing reduces memory usage by 6x compared to Multi-Query Attention and Grouped-Query Attributes. |
| Approach: | They propose a novel approach that extends KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention and Grouped-Query Attributes. |
| Outcome: | The proposed approach reduces KV cache size by 6x with minimal performance loss and scales linearly with model size, batch size, and sequence length. |